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Abstract — The  mismatch  negativity  (MMN),  a  component 
of  event— related  potentials  (ERP),  was  recorded  to  investi¬ 
gate  the  properties  of  the  storage  of  auditory  temporal  se¬ 
quences  in  the  sensory  memory.  First,  the  MMN  elicited 
by  the  infrequent  change  in  a  pair  of  closely  spaced  tones 
was  evaluated.  In  ignored  condition,  it  was  shown  that  the 
MMN  was  elicited  only  when  the  total  duration  of  the  tone 
pair  was  less  than  400ms.  However  in  attended  condition, 
MMN  was  also  elicited  when  the  duration  was  500ms.  This 
result  showed  that  the  function  of  “temporal  integration”, 
which  integrated  the  short  time  history  by  using  a  time 
window,  existed  in  sensory  memory,  and  its  maximum  pe¬ 
riod  was  between  300ms— 400ms,  but  this  period  could  be 
prolonged  by  the  attention  process.  Second,  the  relation¬ 
ship  between  temporal  configuration  of  tone  sequence  and 
elicited  MMN  was  evaluated.  Temporal  configuration  was 
changed  by  inserting  additional  tones  between  the  two  tones 
in  the  tone  pair.  From  this  experiment,  it  was  suggested 
that  the  performance  of  auditory  sensory  memory  was  in¬ 
fluenced  by  the  temporal  configuration  of  the  tone  sequence 
to  be  stored,  as  well  as  its  total  duration.  These  results  may 
help  to  approach  the  neuronal  mechanism  of  auditory  scene 
analysis. 
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I.  Introduction 
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Fig.  1.  Schematic  diagram  of  stimulus  train  used  in  Experiment  1. 
Each  stimulus  consisted  of  a  pair  of  tone  bursts.  Only  the  frequency  of 
the  second  tone  was  different  between  standard  and  deviant  stimuli. 


II.  Experiment  1:  Width  of  Time  Window 

In  Experiment  1,  the  relationship  between  the  total  stim¬ 
ulus  duration  and  the  magnitude  of  elicited  MMN  was  in¬ 
vestigated  to  evaluate  the  period  of  time  window  of  inte¬ 
gration. 


Auditory  sensory  memory  is  thought  to  have  an  impor¬ 
tant  role  for  flexible  processing  in  central  auditory  system, 
as  the  transient  (“buffer”)  memory  storage  of  incoming 
temporal  sequence.  In  the  sensory  memory,  there  exists 
a  function  of  temporal  integration,  which  integrates  suc¬ 
cessive  information  into  some  entities  [1],  [2],  [3],  which  is 
required  for  the  perception  of  auditory  events  [4].  But  the 
period  of  the  time  window  for  temporal  integration,  and 
the  relation  to  the  temporal  configuration  (structure)  of 
the  stored  temporal  information  are  still  remained  unclear. 

Such  properties  of  auditory  sensory  memory  can  be 
observed  by  recording  the  mismatch  negativity  (MMN), 
which  is  one  of  the  event-related  potential  (ERP)  compo¬ 
nents.  MMN  is  elicited  when  some  of  the  frequent  (“stan¬ 
dard”)  stimuli  are  randomly  replaced  by  the  infrequent 
(“deviant”)  stimuli  with  some  different  attributes  to  the 
standard  ones  (oddball  paradigm).  MMN  is  believed  to 
be  the  measure  of  pre-attentive  sensory  memory  proper¬ 
ties  and  is  quite  related  to  the  auditory  discrimination  and 
storage  functions  [5],  [6]. 

In  this  article,  some  experimental  results  on  auditory 
sensory  memory  to  store  tone  sequence  were  shown.  It  was 
shown  that  the  period  of  temporal  window  of  integration 
was  300-400ms,  and  the  attention  could  prolong  the  period 
of  temporal  window  of  integration,  and  that  the  MMN  ac¬ 
tivities  with  attention  had  more  complex  distributions  on 
frontal  area  than  in  ignored  condition.  It  was  also  shown 
the  storage  capacity  of  auditory  sensory  memory  might  be 
related  to  the  temporal  configuration  (structure)  of  stored 
tone  sequence.  These  results  may  be  a  help  to  approach 
neuronal  mechanism  of  auditory  scene  analysis  [4] . 


A.  Methods 

Four  subjects  with  normal  hearing  were  studied  in  an 
electromagnetically  shielded  room.  Auditory  stimuli  were 
delivered  to  the  left  ear  by  headphone.  Each  subject  was 
required  to  read  a  self-selected  book  and  was  instructed  to 
ignore  the  presented  auditory  stimuli  (reading  condition). 

Fig.l  shows  the  stimulation  tone  sequence  in  this  exper¬ 
iment.  Each  stimulus  consisted  of  two  tones,  each  of  which 
was  20ms,  60dB  SL  tone  bursts  (rise/fall  time  was  5ms). 
Frequencies  of  the  first  and  second  tones  were  1.0kHz, 
1.2kHz  (standard  stimulus,  p  =  0.9),  1.0kHz,  1.0kHz  (de¬ 
viant  stimulus,  p  =  0.1)  respectively,  i.e.  the  deviance 
between  standard  and  deviant  stimulus  was  only  the  fre¬ 
quency  of  the  second  tone. 

Inter-stimulus  interval  (ISI)  was  set  to  700ms.  The  stim¬ 
ulus  duration  (Ts)  was  fixed  to  100,  200,  300  and  400ms 
within  each  session,  and  the  relationship  of  Ts  to  elicited 
MMN  was  observed. 

The  EEG  was  recorded  with  Ag-AgCl  electrodes  from 
four  channels  (Fz,  Cz,  F3,  F4).  The  vertical  electro¬ 
oculogram  (EOG)  was  also  recorded  from  the  outer  can- 
thus  of  the  left  eye.  Two  electrodes  attached  on  A1  and 
A2  were  short-circuited  and  took  them  as  the  reference. 

EEG  was  recorded  with  a  bandpass  filter  setting  of  0.1- 
100Hz  and  analyzed  from  100ms  before  the  onset  of  the 
first  tone  to  600ms  after  the  offset  of  the  second  tone  (sam¬ 
pling  rate  1000Hz).  EEG  epochs  in  which  recorded  ampli¬ 
tudes  exceeded  ±100/xV  were  automatically  rejected.  The 
accepted  EEG  epochs  were  averaged. 

For  the  evaluation  of  MMN  magnitude,  the  amplitude 
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(a)  Ts  =  100ms 
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Fig.  2.  [Experiment  1]  Averaged  standard  (dashed)  and  deviant 
(solid)  waveforms  of  one  subject,  (a)  Ts  =  100ms.  (b)  Ta  =  400ms. 


Fig.  3.  [Experiment  1]  Relationship  between  stimulus  duration  (Tg) 
and  amplitude  of  observed  MMN  component  for  all  subjects  (taken 
from  F4).  On  Subjects  A,  C  and  D,  no  significant  MMN  was  observed 
on  Ts  =  400ms  (marked  *). 

of  the  negative  peak  in  the  difference  waveform  (deviant 
—  standard)  whose  latency  was  100-250ms  from  the  onset 
of  the  deviated  tone  was  analyzed  after  the  application  of 
lowpass  filter  (30Hz)  to  the  averaged  waveform.  The  data 
were  statistically  analyzed  using  one-way  analysis  of  vari¬ 
ance  (AN OVA)  for  repeated  measures  with  factors  stimulus 
type  (standard  and  deviant,  p  <  0.05). 

B.  Results 

Fig. 2  shows  the  examples  of  averaged  response  for  stan¬ 
dard  and  deviant  stimuli,  taken  from  F4.  When  the  stimu¬ 
lus  duration  (Ts)  was  100ms,  clear  MMN  could  be  observed 
(Fig. 2(a)).  But  in  this  subject,  no  significant  MMN  com¬ 
ponent  was  appeared  when  Ts  =  400ms. 


The  peak  amplitudes  of  the  observed  MMN  for  all  sub¬ 
jects  are  shown  in  Fig. 3.  This  result  indicated  that  the 
longer  Ts,  the  smaller  the  magnitude  of  MMN.  In  this  ex¬ 
periment,  the  only  deviance  was  the  frequency  of  the  sec¬ 
ond  tone.  So  if  the  two  tones  were  processed  independently 
from  one  another,  such  dependency  on  Ts  would  not  be  ob¬ 
served.  Thus  this  result  might  indicate  that  the  two  tones 
were  processed  as  a  unitary  event,  i.e.  the  temporal  inte¬ 
gration  mechanism  existed  in  auditory  sensory  memory. 

Moreover,  MMN  was  disappeared  for  three  subjects 
when  Ts  =  400ms  (Subjects  A,  C  and  D,  marked  *  in 
Fig. 3).  This  showed  the  time  window  of  integration  could 
be  300  -  400ms  in  ignored  condition. 

III.  Experiment  2:  Attention  Effect 

Attention  effect  for  the  longer  tone  sequence,  for  which 
no  MMN  could  be  observed  on  ignored  oddball  task  was 
studied. 

A.  Methods 

Two  subjects  with  normal  hearing  were  studied  in  an 
electromagnetically  shielded  room.  Auditory  stimuli  were 
delivered  to  the  left  ear  by  headphone.  This  experiment 
was  done  both  in  ignored  (read  self-selected  book)  and 
attended  (count  the  number  of  deviant  stimuli)  condition. 

In  this  experiment,  the  number  of  deviant  stimuli  in 
a  single  session  was  three.  Frequencies  of  the  first  and 
the  second  tones  were  1.0kHz,  1.2kHz  (standard  stimulus, 
p  =  0.85),  1.0kHz,  1.0kHz  (deviant  stimulus  1,  p  =  0.05), 
1.2kHz,  1.0kHz  (deviant  stimulus  2,  p  =  0.05),  1.2kHz, 
1.2kHz  (deviant  stimulus  3,  p  =  0.05),  respectively.  ISI 
was  set  to  700ms,  and  Ts  was  fixed  to  100  and  500ms 
within  each  session.  Note  that  MMN  was  not  elicited  when 
Ts  =  400ms  in  Experiment  1. 

In  this  experiment,  32  channel  whole  head  EEG  signals 
based  on  extended  international  10-20  system,  including 
vertical  and  horizontal  EOG  channels,  were  recorded.  De¬ 
tails  were  the  same  as  in  Experiment  1. 

B.  Results 

Fig. 4  shows  the  averaged  waveforms  for  standard  and 
deviant  1  (deviance  on  the  second  tone)  of  one  subject  for 
both  ignored  and  attended  condition.  In  each  condition, 
maximum  amplitude  of  MMN  was  recorded  from  F4.  In  ig¬ 
nored  condition,  MMN  was  elicited  only  when  Ts  =  100ms 
(Fig. 4(a)).  This  result  followed  the  result  in  Experiment 
1.  However  in  attended  condition,  MMN  was  observed 
when  Ts  =  500ms,  as  well  as  100ms  (Fig.4(b)).  This  result 
showed  that  the  attention  prolonged  the  temporal  window 
of  integration  on  auditory  sensory  memory. 

The  scalp  distributions  of  the  observed  MMN  (shown  by 
arrows  in  Fig. 4)  on  the  same  subject  are  depicted  in  Fig. 5. 
It  was  found  that  the  MMN  activities  in  attended  condition 
(Fig. 5(b),  (c))  had  more  complex  distributions  on  frontal 
area  than  in  ignored  condition  (Fig. 5(a)). 

Though  MMN  is  elicited  both  in  ignored  and  attended 
condition,  MMN  in  attended  condition  may  include  addi¬ 
tional  subcomponents  from  that  in  ignored  condition  [7]. 
Source  localization  and  investigation  on  relation  to  the  at¬ 
tention  and  working  memory  system  [8]  were  left  for  the 
further  study. 
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Fig.  4.  [Experiment  2]  Averaged  standard  (dashed)  and  deviant  (solid)  waveforms  for  Ts  =  100ms  (left)  and  Ts  =  500ms  (right).  Taken 
from  F4  of  one  subject,  (a)  Ignored  condition,  (b)  Attended  condition. 


(a)  (b)  (c) 


Fig.  5.  [Experiment  2]  Scalp  distribution  of  the  observed  MMN. 
Scales  were  from  —5 pV  (black)  to  5//V  (white),  (a)  Ts  =  100ms, 
ignored,  (b)  Ts  =  100ms,  attended,  (c)  Ta  =  500ms,  attended. 


IV.  Experiment  3: 

Temporal  Configuration  of  Tone  Sequence 

In  Experiment  3,  the  relationship  between  capacity  of  au¬ 
ditory  sensory  memory  and  temporal  configuration  (struc¬ 
ture)  of  the  stored  tone  sequence  as  reflected  by  MMN  was 
evaluated. 

A.  Methods 

Temporal  configurations  of  the  presented  tone  sequence 
are  shown  in  Fig. 6.  Each  sequence  consisted  of  the  first 
tone  (open  box),  inserted  tone  (black  box)  and  the  last 
tone  (gray  box).  All  tones  were  1kHz,  60dB  SL  tone  bursts 
(rise/fall  time  was  5ms)  on  standard  stimulus  ( p  =  0.9), 
and  only  the  frequency  of  the  last  tone  was  replaced  by 
2kHz  on  deviant  stimulus  ( p  =  0.1).  The  durations  of  the 
first  and  the  last  tone  were  fixed  to  40ms. 

In  this  experiment,  temporal  structure  of  the  stimulus 


was  controlled  by  changing  the  property  of  the  inserted 
tone  by  the  following  three  ways.  Experiment  3(a):  num¬ 
ber  of  20ms  inserted  tones  (0,  1,  2,  3),  Experiment  3(b): 
duration  of  an  inserted  tone  (20ms,  40ms,  60ms),  Exper¬ 
iment  3(c):  serial  position  of  an  inserted  tone:  interval 
from  offset  of  the  first  tone  to  onset  of  inserted  tone  (20ms 
(Early),  50ms  (Center),  80ms  (Late)).  The  intervals  in  a 
tone  sequence  were  same  in  Experiment  3(a)  and  3(b). 

The  number  of  subjects  were  five  (Experiment  3(a))  and 
three  (Experiment  3(b)  and  3(c)).  In  Experiment  3(a),  two 
additional  experiments  were  done  for  two  subjects  (Sub¬ 
jects  A  and  B).  Detailed  procedure  was  the  same  as  in 
Experiment  1. 

B.  Results 

The  amplitude  of  the  observed  MMN  on  each  experiment 
is  depicted  in  Fig. 7.  In  Experiment  3(a),  MMN  amplitude 
was  reduced  when  the  number  of  inserted  tones  was  in¬ 
creased  on  four  subjects  (Fig. 7(a)).  In  Experiment  3(b), 
MMN  amplitude  was  the  smallest  when  the  duration  of  in¬ 
serted  tone  was  60ms  for  all  subjects  (Fig. 7(b)).  And  on 
two  subjects,  the  largest  MMN  amplitude  was  observed  for 
the  duration  40ms.  And  in  Experiment  3(c),  MMN  ampli¬ 
tude  was  reduced  when  the  inserted  tone  was  shifted  to  the 
last  tone  on  two  subjects  (Fig. 7(c)). 

These  results  suggested  that  the  temporal  configuration 
of  the  stored  tone  sequence  (complexity  of  tone  sequence, 
ratio  between  the  total  periods  of  tones  and  intervals  in 
a  tone  sequence,  and  temporal  pattern  of  tone  sequence), 
as  well  as  the  total  duration  of  the  tone  sequence,  affected 
the  performance  of  the  auditory  sensory  memory.  Detailed 
investigations  were  left  for  the  further  study. 
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Fig.  6.  Tone  sequence  configuration  presented  to  subjects  in  Experi¬ 
ment  3.  On  standard  stimulus,  first  (open  box),  inserted  (black  box) 
and  last  (gray  box)  tones  were  1kHz  tone  bursts,  and  frequency  of  last 
tone  was  replaced  to  2kHz  on  deviant  stimulus.  The  effect  of  change 
in  (a)  the  number  of  inserted  tones  (Experiment  3(a)),  duration  of 
an  inserted  tone  (Experiment  3(b)),  (c)  serial  position  of  an  inserted 
tone  (Experiment  3(c)),  were  investigated. 


V.  Conclusions 

Some  properties  of  auditory  sensory  memory  to  store 
temporal  information  were  shown.  In  Experiment  1,  the 
relationship  between  the  temporal  integration  and  the  du¬ 
ration  of  stored  temporal  sequence  in  ignored  condition 
was  examined.  It  was  shown  that  the  period  of  tempo¬ 
ral  window  of  integration  was  300-400ms.  In  Experiment 
2,  the  attention  effect  for  the  storage  in  auditory  sensory 
memory  was  investigated.  It  was  found  that  the  attention 
could  prolong  the  duration  of  temporal  window  of  integra¬ 
tion,  and  that  the  MMN  activities  with  attention  had  more 
complex  distributions  on  frontal  area  than  in  ignored  con¬ 
dition.  And  in  Experiment  3,  the  influence  of  temporal 
configuration  of  stored  tone  sequence  to  the  MMN  magni¬ 
tude  was  observed.  From  this  experiment,  it  was  shown 
the  storage  capacity  of  auditory  sensory  memory  might  be 
related  to  the  temporal  structure  of  stored  tone  sequence. 
These  findings  might  lead  to  uncover  the  nature  of  tempo¬ 
ral  grouping  of  auditory  temporal  stimuli. 
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observed  on  the  data  marked  *. 


[3]  I.  Winkler,  I.  Czigler,  M.  Jaramillo,  P.  Paavilainen  and  R. 
Naatanen,  “Temporal  constraints  of  auditory  event  synthesis:  ev¬ 
idence  from  ERP,”  NeuroReport ,  9,  pp. 49 5-499,  1998. 

[4]  A.  S.  Bregman,  Auditory  scene  analysis,  Cambridge,  MA:  MIT 
Press,  1990. 

[5]  H.  Tiitinen,  P.  May,  K.  Reinikainen  and  R.  Naatanen,  “Attentive 
novelty  detection  in  humans  is  governed  by  pre-attentive  sensory 
memory,”  Nature ,  372,  pp. 90-92,  1994. 

[6]  S.  Kanoh,  T.  Itoh,  K.  Kazami,  R.  Futami  and  N.  Hoshimiya, 
“A  study  on  mismatch  response  to  a  change  in  tone  duration  by 
EEG  and  MEG,”  Japanese  Journal  of  Medical  Electronics  and 
Biological  Engineering  (in  Japanese),  34,  4,  pp.  128-136,  1996. 

[7]  R.  Naatanen,  M.  Simpson  and  N.  E.  Loveless,  “Stimulus  de¬ 
viance  and  evoked  potentials,”  Biological  Psychology,  14,  pp.53- 
98,  1982. 

[8]  A.  Baddeley,  “Working  memory,”  in  The  Cognitive  Neuro¬ 
sciences,  M.  S.  Gazzaniga,  Ed.,  Cambridge,  MA:  MIT  Press, 
1995,  pp. 755-764. 


